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The objective of this paper is to design performance metrics and respective formulas to quantitatively evaluate the achievement 
of set objectives and expected outcomes both at the course and program levels. Evaluation is defined as one or more processes for 
interpreting the data acquired through the assessment processes in order to determine how well the set objectives and outcomes 
are being attained. Even though assessment processes for accreditation are well documented but existence of an evaluation process 
is assumed. This paper focuses on evaluation process to provide insights and techniques for data interpretation. It gives a complete 
evaluation process from the data collection through various assessment methods, performance metrics, to the presentations in the 
form of tables and graphs. Authors hope that the articulated description of evaluation formulas will help convergence to high quality 
standard in evaluation process. 
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I. Introduction 

A ny educational program starts with a mission statement, 
objectives, and the program or student outcomes. Mission 
statement describes the broad goals of the program. Program 
educational objectives (PEO) are broad statements to describe 
what graduates are expected to attain within a few years of 
graduation. Student outcomes (SO) describe what students are 
expected to know and be able to do by the time of graduation. 
SOs relate to the skills, knowledge, and behaviors that students 
acquire as they progress through the program m. 

Assessment is defined as one or more processes that identify, 
collect, and prepare the data necessary for evaluation. Eval¬ 
uation is defined as one or more processes for interpreting 
the data acquired through the assessment processes in order 
to determine how well SOs are being attained Cl- The 
quality assurance of an academic program depends upon the 
quantitative and qualitative measurements of these PEOs and 
SOs. The main component of the quality assurance system 
is the continuous improvement process. Eaculty and students 
assess the achievement of SOs while advisory board members 
from industry and academia, program alumni, and employers 
are the main resource for accessing the PEO achievements. 
Continuous improvement requires regular documented assess¬ 
ment and evaluation of SOs. These evaluation are used as input 
to make changes in curriculum, teaching methodologies, and 
revision of SOs or PEOs. 

There are many international academic accreditation bodies 
establishing standards, guidelines, and criteria to ensure a 
certain degree of quality assurance and progress towards 
continuous improvement. Accreditation Board for Engineering 
and Technology (ABET) (http;//www.abet.org) is a US-based 
educational programs accreditation body. It is recognized 
as the worldwide leader in assuring quality and stimulating 
innovation in applied science, computing, engineering, and 
technology educations. Its responsibilities include ’’organizing 
and carrying out a comprehensive process of accreditation of 


pertinent programs leading to degrees, and assisting academic 
institutions in planning their educational programs” ffl. At 
the national level in Saudi Arabia, National Commission for 
Assessment and Accreditation (NCAAA) (http://ncaaa.org.sa) 
is responsible for the quality, effectiveness and continuing 
improvement in the quality of post secondary education. The 
presented evaluation methodologies are equally applicable to 
ABET and NCAAA accreditation processes. 

Many authors have published their work on accreditation 
process, continuous improvement, assessment strategies but 
there is no work in the literature that focuses on the evaluation 
of the assessment data at course and program levels. A 
complete procedure for ABET accreditation for Engineering 
programs at Qassim University has been presented in ||2l- It 
gives the detailed implementation of the continuous improve¬ 
ment process, effecting major changes in the educational plan, 
curricular content, facilities, activities, teaching methodolo¬ 
gies, and assessment practices. But this paper does not go 
into the details of evaluation process. In Q authors present 
an assessment plan and continuous improvement process in 
preparation for ABET in department of engineering at James 
Madison University. They have introduced course assessment 
and continuous improvement (CACI) reports at course-level 
and student outcomes summary report (SOSR) at program- 
level. This paper shows some sample reports and assessment 
templates but does not discuss the evaluation process. A web- 
based tool has been introduced in H for outcome-based 
open-ended and recursive hierarchical quantitative assessment. 
This quantitative assessment is used to structure outcomes 
and measures into a leveled hierarchy, with course outcomes 
at the bottom and more general objectives at the top. A 
general curriculum outcome (GCO) layer has been added 
between course’s outcomes and program or student’s out¬ 
comes. In 0 the authors describe the development of an 
Excel spreadsheet and the associated assessment tools for a 
technical design course to measure its success and ensure 
its continuous improvement to meet the requirements of the 
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ABET engineering criteria. In 0 both direct and indirect 
measures are used to collect and analyze data to assess the 
attainments of the student outcomes. To ensure data integrity, 
a set of rubrics with benchmarks and performance indicators 
at both the program and curriculum levels are developed. Each 
outcome has been assessed for different levels (introductory, 
beginning, developing, proficient, exemplary) and from differ¬ 
ent sources. The article Q presents discussions on writing 
learning outcomes and to assess soft skills in engineering 
education. ABET accreditation preparations for construction 
engineering program at the American University in Cairo 
are described in lO. It explains the mapping of department 
mission and objectives with university mission and objectives, 
respectively. Then the PEO to SO mapping and SO to course 
learning outcome (CEO) are provided. The paper ||9l describes 
the assessment techniques and the mapping of CEO to SO 
without the insight of evaluation process. A case study cni 
describes the features that contribute to assessment quality at 
the programme, course and task level. This case study has a 
particular focus on the technical such as task analysis and task 
relationship patterns. Another case study im presents a health 
science program reform and evaluation. It discusses potential 
for evaluation to establish responsive communication between 
students, teaching staff and programme administrators, ensur¬ 
ing a match between the intended, implemented and attained 
curriculum. None of these works however, provide the detail 
of evaluation metrics and their use in course and program 
assessment. 


Fig. 1: Components of direct and indirect approaches in assessment 
process. Direct approach depends on computation and analytical 
analysis of collected data 



This paper presents the evaluation of assessment data ob¬ 
tained from direct and indirect assessments as shown in Eig. [T] 
It gives a complete evaluation process from the data collection 
through various assessment methods, performance metrics, to 
the presentations in the form of tables and graphs. Authors 
hope that the articulated description of evaluation formulas 
will help convergence to high quality standard in evaluation 
process. The rest of the article is organized as follows; the next 
section describes the performance metrics formulation, course 
level and program level evaluation based on performance 
metrics are explained in section III and IV, analysis and 
interpretations are discussed in section V and conclusions are 
drawn in section VI. 


II. Performance Metrics Eormulation 


Let Aij^rn be the marks obtained by student m in question 
j of assessment i (homework, assignment, quiz, midterm, or 
final etc). Here i can take the values, i = 1,2,...,/, j can 
take the values ji = 1,2,..., J^, and m can take the values 
TO = 1, 2,..., M. I, Ji, M are the total number of assessments, 
questions in assessment i, and the students, respectively. 

Eor quantitative analysis, question is a basic unit of compu¬ 
tation for assessment. Average score of question j in assess¬ 
ment i that has M students is given by 


1 

m—l 


( 1 ) 


Bi j can be written in vector form as 

B = [ Bi Ba . . Bj (2) 


Passing threshold (PT) could be absolute, relative or composite 
a, such that the PT of question j in assessment i is given by 
one of the following: 


PT,,, = a X (3) 

PTij = Bij (4) 

PTij = mm{Bij, a X Qlj} (5) 

where Ql°j is the maximum or total marks of question j in 
assessment i and 0 < a < 1. The maximum, minimum, stan¬ 
dard deviation, and x-th percentile of question j in assessment 
i are calculated as 


A- • 

= max Ai j ■ 

m 

A.. 

= min Ai . 

m 


= stdevAij^m 

A ■ ■ 

■P^i^j,per 

= percentile{Aij^,,x) 


Course learning outcomes describe what students are ex¬ 
pected to learn in a course. A mapping between CEO and 
assessment questions is required to compute the attainment 
of the course CEOs. If a course covers N number of CEOs 
then CEO is written as CLOn, n = 1,2,...N. The 
three dimensional matrix A is converted into a two dimension 
matrix A as 

A = [ Af Ai’ . . Af ] (6) 

where Af is the transpose matrix of assessment matrix 
A-jxm- Matrix A has the dimension M x L, where M is the 
total number of students and L — X]i=i Ji ^ total number 
of questions in all assessments. 

CEO to SO mapping matrix is by 


CSi,a 

CSi,b ■ ■ ■ 

CSi,k 

CS2,a 

CS2,b ■■■ 

CSiY-i^k 

CS^.a 

• ■ • 

CSN,k 


The matrix element CSn,a is a variable. CSn,a > 0 if CLOn 
maps to SO a, otherwise CSn,a = 0. A non-zero value is a 
relevance of the CEO to compute the SO. It can take values 
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1,2,3, for low, moderate, and high relevance, respectively. 
Similarly, the CLO to question mapping is given by the 
following matrix 


■ CQip 

CQig 

CQi,l 

CQ2,1 

CQ2,2 

CQn-i,l 

CQn,i 

CQn,l-i 

CQn,l 


Rows of above matrix contain binary variables CQn,i that 
represent the mapping of CLO with question I, where 
I maps to question of an assessment i. CQn,i = 1 if 
CLOn maps to question 1. 

Student marks in assessment questions are used to compute 
how well the students have done and what percentage of 
students have met a certain criteria. Every question contributes 
to one or more CLOs and every CLO contributes to one 
or more student outcomes (SO) as shown in dUl and © 
respectively. 


C. Student Achievement per CLO 

Student Achievement per CLO is defined as the percentage 
of students who are above the expected level as shown in (|5]|. 
Expectation or target is a design parameter, one choice of the 
target could be mm{Bi j,0.7 x i.e., the minimum of the 
average obtained marks and the 70% of the total marks I?). 

It counts the number of students who met the criteria by 
comparing each student marks in a question j of an assessment 
i. If the marks obtained Aij^m are greater than the passing 
threshold PTij, then it increments the counting variable CPS 
(count pass student) by 1. Einally, CPSij or contains 

number of passed students for each question j in assessment i. 
Therefore, the average student achievement per CLO is given 
by 


SA CLOr, = 


EL 


i=l Mi X CQn,i,j X Q\°j 


ELi E-Li QTj X 


(13) 

where Mi is the number of students that participated in the 
assessment i. 


A. CLO Attainment 

This metric is about how well the students have done, in 
percentage, for each CLO. Attainment of a CLO is derived 
from the average marks obtained divided by total marks for 
all questions that maps to the CLO. 

Let be the maximum or total marks of question j in 
assessment i. In general, for assessment i, 

QT = [ Q^i ^^2 • • ^Tj ] (9) 

and 

Then, the percentage of CLO attainment for CLO is given 
by 

attainmentCLOn[%] = E -x 100 (11) 

Eti QE * CQn,i 

The operator * is used for element-wise multiplication. 


D. Student Perception of CLO Attainment 

A course survey is conducted at the end of each semester 
to gauge students’ perception of how well the CLOs were 
covered in the course. It is the average of CLO perception 
from the students. Eor each CLO, students provide their input 
on the scale of 1 — 5 where 1 means CLO is not achieved and 
5 means CLO is achieved completely. Summary of responses 
is given in following matrix. 




SCi,2 

SCi,N ' 


SC = 

SC2,l 

SC2,2 

SCm-i,n 

(14) 


SCm,i 

SCm,n-i 

SCm,n 



Student’s perception of CLO attainment is given by 

1 ^ 

SEJJLOn = ^ I] SCm,n (15) 

771=1 


B. CLO Weightage Information 

In order to get meaningful results, one should design the 
CLOs such that there is a uniform distribution of the marks 
over CLOs in questions to CLO mapping. Eor example, if a 
course contains four CLOs then, ideally, each CLO should get 
25% weightage. 

The ideal case of uniform distribution of marks over the 
CLOs is seldom realized. In these situations, the CLO weigh¬ 
tage information renders a fair picture of % CLO attainment. 
The percentage weightage of CLO is given by 

WeightageCLOn = ^ ^ ^ w{Ai) (12) 

i * 

where w{CQn,j) is the weight of CLOn in question j, w{Ai) 
is the weight of assessment i, and is the total marks of 
assessment i. 


E. x-th Percentile Marks per CLO 

x-th Percentile Marks per CLO is defined as the weighted 
average of x — th percentile marks divided by total marks of 
the questions that map to particular CLO. 

Let xPij be the x-th percentile marks of question j in 
assessment i. In general, for assessment i we have 

xPi = [ xPi 4 xPi _2 ■ ■ xPi^j ] (16) 

and 

xP = [ xPi xP 2 . . xP/ (17) 

Then, the average percentage x-th percentile marks per CLO 
is given by 

xPercentileC LOn = ^ 

EtiQE*CQn,i 

^CPSi is a row vector form of CPSij, similar to e or m 
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F. SO Attainment 

By using CLO-SO mapping in (|7]i, course level SO assess¬ 
ment can be achieved. SO attainment for an SO is computed 
from the CLO attainment of all CLOs that map to the SO. 
SO attainment is defined as the weighted average of CLOs 
attainment (in %). 

attainmentCLOiWi 

attainment SOn = - — (19) 

where C" is the set of CLOs that map to SOn and Wi is the 
weight (or relevance) of mapping between CLO and SO. 


G. Student Achievement per SO 


It is defined as weighted average of student achievement of 
CLOs (in %) that map to a particular SO. 


SA_SOn 


SAJJLOiWi 

Wi 


( 20 ) 


H. Student Perception of SOs Attainment 

Student perception of SOs attainment gives an indirect 
measurement of SO attainment. This metric is derived from 
student perception of CLOs attainment in (flST l and the CLO- 
SO mapping in (O. 


SE_SOn 


SEJJLOiWi 

Wz 


( 21 ) 


I. x-th Percentile per SO 

x-th Percentile per SO uses CLO-Average x-th Percentile % 
marks with CLO-SO mapping in (|7|. 


xPercentileSOn 


xPercentileC LOiWi 
EzgC" w^ 


( 22 ) 


TABLE I: Evaluation of CLOs for the sample course 


CLO 

Attainment 

Achievement 

50th- 

Percentile 

Student 

perception 

1 

79.08 

85 

82.76 

85 

2 

82.91 

91.67 

87.71 

72.5 

3 

78.78 

78.33 

82.97 

72.5 

4 

87.62 

95 

89.92 

67.5 

5 

74.18 

66 

76.44 

77.5 

6 

81.94 

85 

79.48 

77.5 


III. Course Level Performance Evaluation 

Direct assessment of an academic program is performed 
by evaluation of courses in the study plan. If not all, at 
least a selected subset of the courses is required to find out 
program’s success level. Previous section presented formal 
formulations of the performance metrics that can be used in 
course evaluation. This section discusses an implementation 
of these metrics in evaluation of a sample course. The section 
starts with setup required for evaluation followed by evaluation 
results and concludes by discussing issues and concerns. 


A. Course Setup for Evaluation 

Course evaluation is computation of performance metrics 
from basic variable of the course and perform analysis. To 
compute metrics defined in section II from collected data, each 
course must have well defined CLOs, CLO to SO mapping as 
in (|7]), CLO to question mapping in each assessment as in (l8]l, 
and passing threshold as defined in (|5]|. Table [II] shows basic 
variables of a sample course. First two rows show mapping 
between CLOs and questions for all assessments conducted 
in the course. Table HII] shows mapping between CLOs and 
SOs defined by the course designer. A numeric value in a 
cell represents a relationship between a CLO and an SO. A 
value of 1, 2, or 3 indicates that a CLO addresses an SO 
slightly, moderately, or substantively. Passing threshold is set 
to min(at;p, 70%), which is used in computation of student 
achievement per CLO (fTsT i and student achievement per SO 
(Eii. 


B. Performance Evaluation 

This section presents computed values of metrics defined in 
section II for the sample course. Table U presents a summary 
of CLO related metrics and table |IV] shows how to compute 
SO attainments from CLO attainments. 

1) CLO Attainment 

CLO attainment for the sample single section course is 
shown in Fig. |2] CLO attainment quantifies the student attain¬ 
ment level of particular CLO through the percentage marks 
allocated to that CLO. Since this is a percentage value of 
average marks obtained in the questions maps to a particular 
CLO, therefore, it is necessary to either distribute the marks 
uniformly over the CLOs or give an explicit evidence of CLO 
to marks ratio. 


100 


95- 

90- 

85 ^ 



r 



■ 1 
■ ■ 

■ ■ 

■ 

■ 

■1 

■ 

■ 

■ 

■ 


CLOl CL02 CL03 CL04 CL05 CL06 


■ CLO 
Attainment 
I%] 

■ Student 
achievement 
[%] 

■ 50th- 
percentile% 
marks 

■ Student 
perception 
of CLO [%] 


Fig. 2: CLO Performance Evaluation 


2) CLO Weight age 

The CLO weightage for a sample single section course is 
shown in Fig. |3] The CLO attainment and student achievement 
of CLO are based on these weightages. 
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TABLE II: Basic Variables to compute evaluations metrics for a sample course. CLOs and questions mapping (HJ is shown in first two rows 




midterm 


Quizl 

Quiz4 

Quiz3 



HWl 



Quiz2 



Final 



Class participation 

CLOs Covered 

1,2,3 

1 

2 

3,4 

1 

6 

5 

2 

2 

3 

3 

3 

2 

5 

5 

5 

6 

6 

1-6 

Question No. 

1 

2 

3 

4 

1 

1 

1 

1 

2 

3 

4 

5 

1 

1 

2 

3 

4 

5 

1 

Question Marks 

8 

8 

8 

8 

4 

4 

4 

0.4 

0.4 

0.4 

0.4 

0.4 

4 

5 

10 

10 

10 

10 

5 

Actual Average 

5.22 

6.6 

6.99 

6.69 

3.25 

3.664 

3.12 

0.4 

0.4 

0.4 

0.352 

0.128 

3.68 

1.9 

8.3 

7.2 

6.1 

9.3 

4.7 

Passing Threshold (PT) 

5.22 

5.6 

5.6 

5.6 

2.8 

2.8 

2.8 

0.28 

0.28 

0.28 

0.28 

0.128 

2.8 

1.9 

7 

7 

6.1 

7 

3.5 

No. of Students Above PT 

6 

9 

10 

9 

9 

10 

6 

10 

10 

10 

8 

4 

9 

7 

6 

4 

4 

10 

10 

Minimum Marks 

1.2 

4.2 

6 

4.5 

1.3 

3.04 

2.4 

0.4 

0.4 

0.4 

0.16 

0 

2.4 

0 

5 

5 

1 

8 

4.27 

Maximum Marks 

7.35 

7.5 

7.5 

7.5 

4 

3.84 

3.84 

0.4 

0.4 

0.4 

0.4 

0.32 

4 

4 

10 

10 

10 

10 

4.9 

Standard Deviation 

1.78 

0.95 

0.62 

0.92 

0.78 

0.25 

0.47 

0.00 

0.00 

0.00 

0.10 

0.12 

0.44 

1.37 

1.95 

1.60 

2.91 

0.64 

0.21 

50th Percentile Marks 

5.85 

6.75 

7.35 

6.9 

3.3 

3.76 

3.2 

0.4 

0.4 

0.4 

0.4 

0.08 

3.84 

2 

9.5 

6.5 

5.5 

9 

4.79 


TABLE III: Mapping of Course Learning Outcomes (CLOs) to Student Outcomes (SOs) ([7J as Course Assessment Matrix 


CLO 

Course Learning Outcomes Description 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 

Explain the integrated circuit technology and basic engineering process steps of CMOS 



2 









2 

Design combinational/sequential logic gates using CMOS transistor 

2 

1 






1 




3 

Describe MOS circuit design processes, including CMOS design rules, symbolic 
diagrams, and stick diagrams 



3 









4 

Illustrate the details of layout design and verification, concepts of standard gate design 





2 




1 



5 

Describe the VLSI circuit characterization and perfrommace estimation 

3 

2 









2 

6 

Analyze and design complex logic gates in standard CMOS technology 

3 

3 

2 










TABLE IV: Computation of SO attainment from CLO attainment using table [nl] CLO-SO mapping 


CLO 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 



79.08 









2 

82.91 

82.91 






82.91 




3 



78.78 









4 





87.62 




87.62 



5 

74.18 

74.18 









74.18 

6 

81.94 

81.94 

81.94 









SO attainment 

79.68 

79.68 

79.94 


87.62 



82.91 

87.62 


74.18 

Weighted SO attainment 

79.27 

79.52 

79.77 


87.62 



82.91 

87.62 


74.18 

Relevance 

3 

2 

3 


2 



1 

1 


2 



Fig. 3: CLOs Weightage 


3) Student Achievement per CLO 

Student achievement per CLO for the sample single section 
course is shown in Fig. m It is the percentage number of 
students that meet or exceed the target or expectations. There 
is an upper limit for the target (70%) but there is no lower limit 
and it depends upon the average marks. We can get absolute 
student achievement by fixing the target, for example, with 
target value of 60%. 

4) 50-th Percentile per CLO 

The 50-th percentile for the sample course is shown in Fig. 
121 It shows the percentage median marks for each CLO. 


5) Student Perception of CLOs Attainment 

For each CLO, student perception of CLO attainment can 
be on the scale of 1 to 5, 1 mean strongly disagree to 5 for 
strongly agree. Student perception of CLO attainment for the 
sample course is shown in Fig. |2l There were 10 students but 
8 participated in the course survey. 

6) SO Attainment 

Bar graphs for SO attainment are shown in Fig. |4] These 
levels are averages of CLO attainments that map to particular 
SO, therefore, the health of CLO attainments and CLO-SO 
mapping is critical. 



7) Student Achievement per SO 

Student achievement for the sample course is given in Fig.|4] 
This is a derived value from CLO achievements and depicts the 
percentage number of students achieved the set target averaged 
over the CLOs mapped to that SO. 
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8) 50-th Percentile per SO 

The 50-th percentile per SO values are derived from 50- 
th percentile per CLO. Fig. |4] depicts the 50-th percentile or 
median marks for each mapped SO. 

9) Student Perception of SOs Attainment 

The student perception of SOs attainment is shown in Fig. 
m It is an indirect measurement obtained from the course exit 
survey where students provide their feedback about the CLOs 
attainment. 

C. Issues and Guidelines 

Course designer is responsible to establish quality mapping 
between CLOs and SOs. Course instructor is responsible for 
CLOs to questions mapping for all assessments. Quality of 
these mappings have direct impact on the evaluation results as 
discussed in rest of this section. 

1) Relationship of Questions, Marks distribution and CLOs 

It has been observed that questions to CLO mapping re¬ 
quires uniform marks distribution over the CLOs. The quan¬ 
titative measurement of CLOs provides the baseline data for 
direct assessment, therefore, questions to CLOs mapping is 
critical in direct assessment. CLOs should be designed in such 
a way that they cover all the core topics (qualitative equality) 
and course assessments should cover all CLOs with uniform 
marks distribution over the CLOs (quantitative equality). Sim¬ 
ilar measures are required in capstone project rubrics’ design. 
Capstone project is an important entity of program in which 
students apply the knowledge gained during the course of 
the program to solve the engineering problems. The capstone 
project rubrics map to CLOs and these CLOs usually cover 
all the SOs. Since the sample size in this assessment is not 
as large as of direct assessment therefore results may differ in 
these assessments. 

2) Questions to CLOs Mapping Approaches 

Due to the many-to-many mapping between questions and 
CLOs, a common question arises about the weights of a ques¬ 
tion that maps to multiple CLOs. There are three possibilities; 

• One-to-many mapping with equal weights 

> One-to-many mapping with proportional weights 

■ One-to-one mapping between questions and CLOs. 

In this manuscript, equal weights have been used in questions 
to CLOs mapping. The proportional weights add one more 
level of complexity for the faculty and hence more chances 
of errors. One-to-one mapping is another attractive solution 
which eliminates the weight problem because in this case 
one question can be mapped to one CLO at most. In this 
technique many questions can be mapped to one CLO but 
converse is not possible. Proportional weights and one-to-one 
schemes require a proper design of CLOs and the mapping 
table between questions and CLOs. 

3) CLOs to SOs Mapping within a Course 

There are three choices; 

> One CLO can be mapped to any number of SOs without 
weights (one-to-many mapping without weights) 

> One CLO can be mapped to any number of SOs with 
weights (one-to-many mapping with weights) 


■ One CLO can be mapped to one SO only (one-to-one 
mapping) fill 

In this manuscript, one-to-many mapping with weights has 
been used as shown in Table m One-to-many mapping 
without weights assumes equal weights across all SOs mapped 
to a particular CLO. A straight forward way of mapping is 
one-to-one mapping which does not require weights but again 
the design of CLOs is important in this case. 

IV. Program Level Performance Evaluation 

Achievement of student outcomes (SO) and program educa¬ 
tional objectives (PEO) are the basis for program assessment. 
PEOs are computed from prgram level SOs and indirect 
methods of assessment such as graduate exit surveys, alumni 
surveys, and employer surveys El. 

To measure the achievement level of SOs, direct as well 
as indirect methods are used as shown in [T] Previous sec¬ 
tion presents direct assessment approaches and results of SO 
achievements derived from students’ performance in courses. 
These course level SOs achievements are used to build the 
program level direct assessment of SOs. Direct method uses a 
subset of courses in the study plan to compute program level 
SOs. 

TABLE V: Relationship between the Program Educational Objectives 
and Student Outcomes 


PEOs 



student Outcomes 


a 

b 

c d 

e 

f g h 1 

j k 

I 

X 

X 

X X 

X 


X 

II 

X 

X 

X 

X 

X 

X 

III 



X 


X 


IV 





X XX 

X 


A. Program Setup 

The program educational objectives are well-defined and 
published objectives of an institution. Department of Computer 
Engineering at Taif University has published the following 
program educational objectives. 

Within two to three years of graduation, our Computer Engi¬ 
neering Graduates will; 

(I) Be employed as computer engineering professionals 
in public or private sector and/or perform at a satis¬ 
factory level in graduate programs. 

(II) Possess the necessary knowledge in fundamental the¬ 
ories, techniques and tools to solve computer engi¬ 
neering problems, design hardware/software systems 
and improve their performance. 

(III) Demonstrate communication and inter-disciplinary 
collaboration skills, and leadership qualities. 

(IV) Engage in lifelong learning activities within a pro¬ 
fessional, legal and ethical framework. 

A mapping between PEOs and SOs is required to derive 
PEO achievment values from program level SO values. This 
mapping should be designed by stakeholders responsible for 
program assessment. Table |V] present a sample mapping be¬ 
tween PEOs and SOs, which is used in this paper. 






7 


TABLE VI: Program Assessment Matrix based on Courses (Substantial: 3, Moderate: 2, Slight: 1, None: blank) 


Assessment 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

DIRECT and INDIRECT 

English 







3 


2 



Calculus 1,11,111 

3 










2 

Linear Algebra 

3 










2 

Ditterential Equations 

3 










2 

Probability and Stats 

3 











Discrete Math 

3 











Physics 1,11 

3 

3 










Computer Programming 1, 11 

3 

3 

2 






2 

2 


Data Structure 


2 








3 


Operating Systems 

3 

2 







2 

1 


Fundamental ol Database 


3 

2 






2 



Software Engineering 


3 

2 






2 

2 


Digital Logic Design 

3 

3 

I 


I 







Electric Circuits 

3 

3 

2 


I 






3 

Electronics 

3 

3 










Computer Architecture 

2 

3 

I 






1 

I 


Computer Organization and Design 

1 

2 

I 





1 


1 


Microprocessor 

3 

2 

3 








2 

Professional Ethics 



2 



3 






Embedded Systems 

2 

2 



1 






1 

Signals and Systems 

3 

3 

3 


I 







Communication Systems 

3 

1 

2 









Digital Signal Processing 

2 

3 

2 








1 

Control Engineering 

3 

3 

I 


1 







VLSI De.sign 

3 

2 

3 


2 



I 

1 


2 

Capstone Design Project 

2 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 


TABLE VII: Program Assessment Matrix based on Surveys (Substantial: 3, Moderate: 2, Slight: 1, None: blank) 


Assessment 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

INDIRECT 

Graduate Exit Survey 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

Student Course Exit Survey 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

Capstone Project Rubrics 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

Course Assessment by Instructor(s) 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

Course Assessment by Reviewer(s) 

3 

3 

3 

2 

2 

2 

2 

2 

2 

3 

3 


Alumni Survey 

I 

1 

I 

1 

2 

2 

2 

2 

2 

3 

3 

Employer Survey 

I 

1 

I 

1 

2 

2 

2 

2 

2 

3 

3 

Advisory Board Feedback 

1 

1 

I 

1 

2 

2 

2 

2 

2 

3 

3 


Student outcomes are the defined skills that a student should 
have; after successfully completing the educational program. 
An educational program consists of general knowledge, foun¬ 
dation core, advanced core, and elective courses. All courses 
in a program can be used to derive SO achievement levels 
or a set of witness courses can be selected to ensure that 
all SO are properly covered. Required advanced courses in 
the major field of study are obvious candidates for this 
set. Courses such as mathematics, physics, chemistry, and 
English might be included as long as they consistently address 
outcomes. Elective courses or courses whose content varies 
from one offering to another (so that the outcomes might not 
be addressed in a particular offering) should not be included 

II3. 

Core courses of a program include courses related to the 
specific program along with math and basic sciences courses. 
All core courses must have mapping between CEOs and SOs 
ini. Electives courses for which CEOs are not consistent 
in subsequent offering, should not be considered in program 
assessment. 

A program assessment matrix (PAM) can be constructed as 
shown in Table |VT] for direct assessment based on students’ 
performance in the courses. Eor courses to SOs mapping, each 
course gives a relevance number (1 — 3) for each mapped SO in 


the Table |VT] This is calculated from the weights along each 
SO column in Table [111] It has been observed that program 
assessment Table |Vl] should contain the weights for credit 
hours (1—4) along the SOs columns, such that weight is 
one for one credit hour, two for two credit hours and so on. 
It will provide more fair values of attainments, achievements, 
and percentiles. 

An identical matrix should also be used for indirect as¬ 
sessment based on course exit survey that shows students’ 
perception of SO achievement. Surveys are indirect assessment 
methods and a PAM can be created as shown in Table I VII I 
Eirst five rows of the table are related to the course offerings 
in a program while last three row are about perception from 
the stakeholders in the marketplace about the program itself. 
Entries of 1, 2, and 3 for each SO in the matrix denote slight, 
moderate, and substantial relevance to the program outcomes 
lISl . These matrices provide a brief summary of how SOs are 
assessed when attempting to raise the attainment level of a 
particular SO. 

Program level direct assessments are obtained by putting the 
weighted average of SO attainment values from all course level 
SO attainment tables in program assessment Table |VT] Eor 
example, the weighted average of SO attainment (second last 
row) from Table |IV] in the corresponding course row of Table 












































































IVTl Similar matrix can be used to compute student perception 
of SOs for each course as indirect assessment. Since capstone 
project contains both direct and indirect assessment of CLOs 
and SOs, SO attainment can be directly used here. 

B. Performance Evaluation 

There are multiple sources of data to compute the attainment 
of SOs at program level. Bulk of the data and evidences 
come from courses offered during the program and students’ 
performance in the courses. For every course students provide 
their perception on attainment of SOs using course survey. 
Assessment of capstone graduation project is done directly 
based on students’ performance as well as done indirectly 
based on rubrics. 

Table I VllIl shows attainment trend of SO ”a” from different 
direct and indirect assessment methods and same data is shown 
in Fig. 13 The intra-assessment trend for direct assessment 
shows marginal improvement in three years so as in the 
case of capstone project rubrics. But there is significant 
increase in students’ perception of SO attainment. This all 
due to the increasing standards of education quality during 
the ABET preparation and after the ABET accreditation. The 
inter-assessments comparison shows that direct assessment 
levels are constant despite the monotonic increase in stu¬ 
dents’ perception. It exhibits the gradual increase of quality 
standards, realized by students. The capstone project rubrics 
assessment have highest values among all assessments which 
are always anticipated because of small sample sizes and 
for the graduating students. Attainment of other SOs can be 
computed and presented similarly. 

TABLE VIII: Comparison of SO ”a” attainment 

Assessment type 2011-12 2012-13 2013-14 

Direct assessment 66.75 68.9 68.89 

Course exit survey (indirect) 62.98 75.67 83.77 

Capstone rubrics (hybrid) 85.4 82.58 83.36 



Direct assessment Course exit survey Capstone rubrics 
(indirect) (hybrid) 


Fig. 5: Comparison of SO ”a” attainment 
TABLE IX: Attainment of Program Educational Objective I 


Assessment type 

2011-12 

2012-13 

2013-14 

Direct assessment 

70 

68.5 

67 

Graduate exit survey 

76 

80 

83 

Alumni survey 

89 

86 

87 

Employer survey 

79 

85 

84 



Direct Graduate exit Alumni survey Employer survey 

assessment survey 


Fig. 6: Attainment of Program Educational Objective I 

The trend of PEOs attainment can be represented for direct 
assessment, graduate exit survey, alumni survey, and employer 
survey. As an example Table |I3 shows attainment of PEO I 
and a graph representation is shown in Eig. |3 Similar process 
can be used to compute and present attainment of PEO II, III, 
and IV for the program. 

V. Analysis and Interpretations 

This section provides a detail analysis of course and pro¬ 
gram level evaluations based on the formulated evaluation 
metrics. The implementation of these metrics have revealed 
several new directions and interpretations. 

A. Analysis and Interpretations: Course level 

Course evaluation produces quantitative values of attain¬ 
ment, achievement, and x — th percentile metrics for CLOs 
and SOs. Eor a course, relative values of these metrics provide 
insight into what happened in the course and zero value for a 
metric indicates that topics related to the corresponding CEO 
or SO are either not covered in the course or data was not col¬ 
lected for evaluation. Eor a multi-section course, these metrics 
can point to lack of coordination among course instructors, 
and difference in teaching and evaluation standards. 

If a course instructor does not cover some CLOs then 
corresponding metrics values will be zero as shown in figure 
fin and [12] Eor the sample course, CEO 4 and 5 are not 
covered and since these two CLOs maps to SO ”a”, so 
the SO is also not covered in the course. Eor multi-section 
courses, a zero value for any of the defined metrics in some 
of the section indicates lack of coordination among course 
instructors. Eigures [3 and |9| points to different teaching and 
evaluation criteria among different instructors of the same 
course. 

In order to explain the relationship, CEO attainment, student 
achievement, and 50-th have been plotted against the students’ 
average marks in Eig. |7| These graphs show the three evalu¬ 
ation metrics’ values for a range of average marks associated 
with a particular CEO. In this figure, the average marks 
are obtained from normal distribution mean (average) with 
standard deviation 5. Number of students is 30 and the results 
are averaged over 1000 iterations. Erom this figure, it can be 
seen that CEO attainment is a linear function of questions’ 
average marks mapped to that CEO. The attainment is equal 
to the 50-th percentile because the mean and median of normal 
distribution are equal. The composite student achievement 
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Fig. 7: Comparison of CLO attainment, student achievement, and 
50-th percentile 

remains almost constant up to 70% average marks due to the 
passing threshold mm{averageMarks,70%TotalMarks). 
For normally distributed marks, there are always 50% students 
below the average value and 50% students are above the 
average value, hence, student achievement remains constant 
at 50% value. When the average marks go above 70%, the 
passing threshold shifts from average value to 70% and all 
the students with marks greater than 70% contribute to the 
student achievement. At about 80% average marks, the student 
achievement reaches to 100% value because all the students 
even with the standard deviation of 5 now lie above 70% 
threshold. The relative achievement always remains around 
50%. The absolute achievement for a = 0.7 crosses the 50% 
value at average marks equal to 70. 

The designed evaluation metrics give comprehensive results 
when considered collectively as shown in hgures [8] to [12] 

1) Attainment and Achievement: The relationship between 
attainment and student achievement (absolute, compos¬ 
ite) is linear for average marks greater than set target, 
i.e., for sufficiently large population size and normal 
distribution of obtained marks, high values of attainment 
corresponds to high values of achievements and the 
low attainment expects low achievement. Attainment 
and achievement are independent for achievement level 
below the threshold. If the distribution is not normal 
then linearity is not guaranteed. For example, if there 
are 10 students and 9 students secure 50 marks (out of 
100) and one student get 10 marks then the composite 
achievement is 90% but the attainment is 46% 

2) Attainment and Percentile: The 50-th percentile (or 
median) gives an additional information about the health 
of attainment. It is also called location parameter. Me¬ 
dian close to attainment indicates normal distribution of 
marks. 

3) Achievement and Percentile: If 50-th percentile (median) 
is equal to the target value of achievement then the 
achievement is equal to 50%. Median values above the 
the achievement target shows that more students have 
met the expectation and value of achievement will be 
high. Median values less than the achievement target 
results in the achievement level less than 50%. 


4) Attainment, Achievement, and Percentile: Attainment 
and 50-th percentile (median) have the same units, i.e., 
average and median marks in a particular CLO, whereas, 
student achievement gives the number of students. If 
attainment and median are similar (normal distribution) 
and have high values then the absolute and composite 
achievements will also be high because more number of 
students would have marks greater than the set target, 
whereas, the relative achievement will remain flat at 
50% because of normally distributed marks. Conversely, 
if attainment and median have low levels then the 
composite achievement becomes 50%. Note that the 
absolute achievement is proportional to the attainment 
and median near the target and becomes independent 
for the average marks sufficiently less or greater than 
the target value. 



Fig. 8: Comparison of computed metrics for three sections of the 
same course for a CLO-SO mapping 



Fig. 9: Comparison of computed metrics for four sections of the same 
course for a CLO-SO mapping 


B. Analysis and Interpretations: Program level 

Program assessment depends mainly on the course level 
direct assessments. Other sources of program assessments are 
indirect assessments through surveys which complement the 
results obtained from direct assessments. 

SOs and PEOs level obtained from various sources are 
hnally analyzed through statistical tool. Results from two 
datasets are compared with t-test whereas, more than two 
datasets are compared with analysis of variance (ANOVA) test 
and graphical plots IITtII . 
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Fig. 10: Relationship of SO attainment and achievment 



I b j 


Fig. 11: Relationship of SO attainment and achievment and coverage 
of SO in the course 



Fig. 12: CLO metrics and coverage of CLO in the course 


TABLE X: Interpretations of attainment, student achievement, and 
median values. Other combinations are not common 


Attainment 

Student 

Achieve¬ 

ment 

50th- 

Percentile 

(Me¬ 

dian) 

Interpretations 

High 

High 

High 

If values are greater than 
80, then course and as¬ 
sessments should be re¬ 
viewed 

Low 

High 

Low 

Majority have low grades 
and few have worse; spe¬ 
cial case of one student 
with low grade 

Low 

Low 

Low 

Majority have low grades 
and few have good 


The indirect assessments of SOs and PEOs are very subjec¬ 
tive. One cannot compare the SO’s and PEO’s levels in direct 
and indirect assessment using the same weights. The indirect 
assessments, especially, the surveys, require extra efforts in 
determining the the weights relative to the direct assessment. 
These weights depend upon the type of employer, quality 
of advisory board, and the interaction level with alumni. In 
general, these weights may differ institute by institute and 
region by region. 


VI. Conclusions 

Established universities have assessment and evaluation pro¬ 
cesses in-place but for less established and newer institutions 
learning, understanding, and establishing these processes are 
big challenges. This paper describes a complete evaluation 
process of assessment data for an educational degree program 
with the help of various evaluation metrics and quantification 
approaches of PEOs and SOs attainment levels. The main 
focus of this paper is the formulation and implementation of 
performance metrics for CEOs, SOs, and PEOs evaluation. 
Analysis and interpretation of processed data along with the 
relationship among questions, CEOs and SOs is presented to 
give an insight into evaluation process. 
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